In this notebook, we will present how to classify flower images by using transfer learning from a pre-trained network.
A network model that has been saved and earlier trained using a sizable dataset is known as a pre-trained model.
The concept behind transfer learning for image classification is that if we use a model that was trained on a really big, representative dataset, this model can serve as our basic model for categorising images. This allows to save a lot of training time by utilizing the feature maps.
Here, we're gonna test two approaches:
feature extraction: We extract significant features from fresh data using representations learned by a network that has already undergone training. Simply adding a new classifier will allow us to use the feature maps that have already been learned for the dataset and will be trained on top of the pretrained model. We do not need to restart this trained network, of course. There are already features in the basic convolutional network that are generally helpful for classifying images. It should be noted that the pretrained model's ultimate classification component only considers the initial classification.
fine tuning : This approach involves unfreezing a few of the top layers of the previously frozen model and jointly training the new top layer (to identify our particular datasets) and these final layers of the frozen model. We take this action because the first convolution layers of the base model only extract fundamental features (edges, vertical/horizontal lines, etc.), while the final layers of the base model extract top-level feature maps. By fine-tuning the base model's top feature representation, we can increase their specificity for our classification job.
# import stuff
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import seaborn as sns
from PIL import Image
from IPython.display import Image, display
import os
from sklearn.metrics import confusion_matrix
from sklearn.datasets import load_files
from sklearn.model_selection import train_test_split
import tensorflow_hub as hub
from keras.utils import to_categorical
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from tensorflow.keras.optimizers import Adam, RMSprop, SGD
from tensorflow.keras.applications import ResNet50, VGG16, VGG19, MobileNetV2
from tensorflow.keras.applications.resnet50 import preprocess_input as prepro_res50
from tensorflow.keras.applications.vgg19 import preprocess_input as prepro_vgg19
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, Dense, Flatten, GlobalAveragePooling2D, BatchNormalization, Dropout, MaxPool2D, MaxPooling2D
# load the backend
from keras import backend as K
# prevent Tensorflow memory leakage
K.clear_session()
In this notebook, we'll make use of a much complex architecture with the MobileNetV2 model. It has been developed in 2018, MobileNetV2 is a significant improvement over MobileNetV1 and pushes the state of the art for mobile visual recognition including classification, object detection and segmentation. If you're interested, you can find more details in the release paper.
path_data = r'D:\T431 - George Brown\DL Math 2\Flower_Classication_using_Transfer_Learning\flowers'
print(os.listdir(path_data))
['daisy', 'dandelion', 'rose', 'sunflower', 'tulip']
from os.path import join
img_folders = [join(path_data, folder) for folder in os.listdir(path_data)]
list(img_folders)
['D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\daisy', 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\dandelion', 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\rose', 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\sunflower', 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\tulip']
data_dir = r'D:\T431 - George Brown\DL Math 2\Flower_Classication_using_Transfer_Learning\flowers'
data = load_files(data_dir, random_state=28, shuffle=True)
X = np.array(data['filenames']) # files location of each flower
y = np.array(data['target']) # target label of each flower
labels = np.array(data['target_names'])
# remove eventual .pyc or .py files
pyc_file = (np.where(file==X) for file in X if file.endswith(('.pyc','.py')))
for pos in pyc_file:
X = np.delete(X, pos)
y = np.delete(y, pos)
print(f'Data files - {X}')
print(f'Target labels - {y}') # numbers are corresponding to class label,
# we have to change them to a vector of 5 elements
print(f'Name labels - {labels}')
print(f'Number of training files : {X.shape[0]}')
Data files - ['D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\rose\\2265390547_2409007cef_n.jpg' 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\dandelion\\34689593326_0fd3fbc38a_n.jpg' 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\tulip\\10094729603_eeca3f2cb6.jpg' ... 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\dandelion\\5598845098_13e8e9460f.jpg' 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\tulip\\5757091018_cdfd79dfa6_m.jpg' 'D:\\T431 - George Brown\\DL Math 2\\Flower_Classication_using_Transfer_Learning\\flowers\\dandelion\\34537877932_f9a3476a4d_n.jpg'] Target labels - [2 1 4 ... 1 4 1] Name labels - ['daisy' 'dandelion' 'rose' 'sunflower' 'tulip'] Number of training files : 4317
# Flower species number
df = pd.DataFrame({'species': y})
print(df.shape)
df.head()
(4317, 1)
| species | |
|---|---|
| 0 | 2 |
| 1 | 1 |
| 2 | 4 |
| 3 | 2 |
| 4 | 4 |
# associate names to species number
df['flower'] = df['species'].astype('category')
df['flower'].cat.categories = labels
df.head()
C:\Users\pinku\AppData\Local\Temp\ipykernel_10052\1180994331.py:3: FutureWarning: Setting categories in-place is deprecated and will raise in a future version. Use rename_categories instead. df['flower'].cat.categories = labels
| species | flower | |
|---|---|---|
| 0 | 2 | rose |
| 1 | 1 | dandelion |
| 2 | 4 | tulip |
| 3 | 2 | rose |
| 4 | 4 | tulip |
Let's check how many of each species of flowers are present.
fig, ax = plt.subplots()
ax = sns.countplot(x="flower", data=df)
ax.set(ylabel='Count', title='Flower species distribution')
ax.tick_params(axis='x', rotation=15)
Now, we're gonna load the different images and transform them into numpy arrays.
image_size = 224 # standard value for Transfer learning usecase (MobileNet, ResNet50, VGG16, VGG19)
def read_and_prep_images(img_paths, img_height=image_size, img_width=image_size):
imgs = [load_img(img_path, target_size=(img_height, img_width)) for img_path in img_paths] # load image
img_array = np.array([img_to_array(img) for img in imgs]) # image to array
return(img_array)
X = np.array(read_and_prep_images(X))
print(X.shape) # (4323, 224, 224, 3) = (num_images, height_size, width_size, depth=RGB)
(4317, 224, 224, 3)
# Let's have a look at 6 randomly picked flowers.
N = 18 # flowers to display
fig, axes = plt.subplots(3, 6, figsize=(16,6))
for ax, j in zip(axes.flat, np.random.randint(0, len(X), N)):
ax.imshow(X[j].astype(np.uint8))
ax.set(title=f'Flower: {labels[y[j]]}', xticks=[], yticks=[])
fig.tight_layout()
num_classes = len(np.unique(y))
print(f'Number of classes: {num_classes} --> {labels}')
Number of classes: 5 --> ['daisy' 'dandelion' 'rose' 'sunflower' 'tulip']
Labels are the 5 species number (from 0 to 4). Thus, we need to encode these labels to one-hot vectors. For instance, an image of a sunflower should have a label 3 and a corresponding y = [0,0,0,1,0].
y = to_categorical(y, num_classes)
print(y.shape)
(4317, 5)
Here, we're gonna split our dataset into a training, a validation and a testing one. This ensures that there are no bias: the model is trained on images with known labels, then we test our model accuracy on the validation dataset on images that our model did not see before. Finally, we compute the accuracy on the test dataset.
#train, validation and test from the train dataset
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, shuffle=True,
test_size=0.25, random_state=28)
Xval, Xtest, yval, ytest = train_test_split(Xtest, ytest, test_size=0.5,
shuffle=True, random_state=28)
print(f'Train dataset: {Xtrain.shape[0]}')
print(f'Validation dataset: {Xval.shape[0]}')
print(f'Test dataset: {Xtest.shape[0]}')
Train dataset: 3237 Validation dataset: 540 Test dataset: 540
# release memory
del X
del y
When our model will be built, we need to specify an accuracy function, a loss function and an optimisation algorithm.
The accuracy function is used to evaluate the performance of the model.
The loss function is used to measure how the model performs on data with known labels. It tells us how poorly the model performs in a supersised system. For multi-label classification, we make use of a specific loss function called as categorical_crossentropy (similar to cross-entropy in maths).
Finally, the optimizer function is used in order to minize the loss function by changing model parameters (weighs values, filters kernel values etc.).
For this classification problem, we choose the RMSprop optimizer which is very efficient and commonly used (more details on the optimizers on Keras here).
Since deep networks can take quiet a time for the optimizer to converge, we're gonna use an annealing method of the learning rate (LR).
The LR is basically the step by which the optimizer is 'walking'. A hight LR correspond to big steps and thus the convergence is faster. However, in that case the sampling is not really efficient since the optimizer do not fall especially in the right minima.
At the opposite, have a low LR means that the optimizer will probably find the right local minima but it will take a lot of time.
The idea here is to start from a low value but not so low and then decrease the LR along the training to reach efficiently the global minimum of the loss function. Using the ReduceLROnPlateau method , we are able to choose to reduce the LR by a coefficient (here 75%) if the accuracy has not improved after a number of epochs (here 3).
In addition, we use the EarlyStopping method to control the training time: if the accuracy has not improved after 5 epochs we stop.
Finally we make use of the ModelCheckpoint which is useful for monitoring the best found weights during the training.
For now, we're doing feature extraction i.e. we freeze the convolutional base (MobileNet). Then, we add a classifier on top of it and train this top-level classifier.
# Load the VGG19 model without the final layers (include_top=False)
img_shape = (image_size, image_size, 3)
print('Loading MobileNetV2 ...')
base_model = MobileNetV2(input_shape=img_shape,
include_top=False,
weights='imagenet')
print('MobileNetV2 loaded')
base_model.trainable = False
#base_model.summary()
Loading MobileNetV2 ... MobileNetV2 loaded
base_model.output_shape
(None, 7, 7, 1280)
Now, we need to generate predictions from the block of features, average over the spatial locations, using a GlobalAveragePooling2D layer to convert the features to a single 1280-element vector per image. Finally, we'll some regular Dense layer with a final one with 5 units corresponding to each species of flower.
model = Sequential([base_model,
GlobalAveragePooling2D(),
Dense(num_classes, activation='softmax')
])
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
mobilenetv2_1.00_224 (Funct (None, 7, 7, 1280) 2257984
ional)
global_average_pooling2d (G (None, 1280) 0
lobalAveragePooling2D)
dense (Dense) (None, 5) 6405
=================================================================
Total params: 2,264,389
Trainable params: 6,405
Non-trainable params: 2,257,984
_________________________________________________________________
Note that only ~ 6000 parameters will be trained, the other ~2.2M from the MobileNetV2 model were already trained.
# callbacks
weight_path = '{}_best_model.h5'.format('flower')
checkpointer = ModelCheckpoint(weight_path,
monitor='val_accuracy',
verbose=1,
save_best_only=True,
mode='auto',
save_weights_only=True)
# set a learning rate annealer
learning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy',
patience=3,
verbose=1,
factor=0.7,
min_lr=0.00001)
# early stop if not improvement of accuracy after 5 epochs
early = EarlyStopping(patience=6,
verbose=1)
callbacks = [checkpointer, learning_rate_reduction] #, early]
# Optimizer
opt = RMSprop(learning_rate=0.001, rho=0.9, epsilon=1e-08, decay=0.0)
# Compilation
model.compile(optimizer=opt,
loss='categorical_crossentropy',
metrics=['accuracy'])
A useful trick to ovoid any overfitting is to use data augmentation. What is that? Well, the idea is to add artificially data into our dataset. But of course not any data, we alter the dataset with tiny transformations to reproduce very similar images.
For instance, we rotate of a few degree an image, we de-center it or we zoom in or out a little bit. These common augmentation techniques are horizontal/vertical flips, rotations, translations, rescaling, random crops, adjust brightness and more.
Thanks to these transformations, we can get bigger dataset (x2, x3 in size) and then train our model in a much robust way.
image_size = 224
batch_size = 32
path = r'flowers'
#train_gen = train_aug.flow(Xtrain, ytrain, batch_size=batch_size)
# The validation data must not have data augmentation
#valid_gen = valid_no_aug.flow(Xval, yval, batch_size=batch_size)
train_datagen = ImageDataGenerator(
rescale=1./255, # rescale pixel values [0,255] to [0,1]
horizontal_flip=True, # random horizontal flip
width_shift_range=0.2, # random shift images horizontally (fraction of total width)
height_shift_range=0.2, # random shift images vertically (fraction of total height)
zoom_range=0.2) # random zoom image
#rotation_range=20, # random rotation
#shear_range=0.2) # shear transfo
#validation_split=0.2) # splitting train / test datasets
test_datagen = ImageDataGenerator(
rescale=1./255)
#validation_split=0.2)
train_gen = train_datagen.flow(
Xtrain, ytrain,
batch_size=batch_size,
shuffle=False) # already applied
valid_gen = test_datagen.flow(
Xval, yval,
batch_size=batch_size,
shuffle=False)
batch_size = 32
epochs_0 = 10
steps_per_epoch = len(train_gen.x) // train_gen.batch_size
validation_steps = len(valid_gen.x) // valid_gen.batch_size
history = model.fit(
train_gen,
steps_per_epoch=len(Xtrain) // batch_size, # or batch_size=32
epochs=epochs_0 ,
validation_data=valid_gen,
validation_steps=len(Xval) // batch_size,
callbacks=callbacks)
Epoch 1/10 101/101 [==============================] - ETA: 0s - loss: 0.8242 - accuracy: 0.6927 Epoch 1: val_accuracy improved from -inf to 0.80273, saving model to flower_best_model.h5 101/101 [==============================] - 39s 293ms/step - loss: 0.8242 - accuracy: 0.6927 - val_loss: 0.5514 - val_accuracy: 0.8027 - lr: 0.0010 Epoch 2/10 101/101 [==============================] - ETA: 0s - loss: 0.4731 - accuracy: 0.8365 Epoch 2: val_accuracy improved from 0.80273 to 0.81250, saving model to flower_best_model.h5 101/101 [==============================] - 27s 266ms/step - loss: 0.4731 - accuracy: 0.8365 - val_loss: 0.5507 - val_accuracy: 0.8125 - lr: 0.0010 Epoch 3/10 101/101 [==============================] - ETA: 0s - loss: 0.4072 - accuracy: 0.8546 Epoch 3: val_accuracy improved from 0.81250 to 0.84570, saving model to flower_best_model.h5 101/101 [==============================] - 26s 258ms/step - loss: 0.4072 - accuracy: 0.8546 - val_loss: 0.4689 - val_accuracy: 0.8457 - lr: 0.0010 Epoch 4/10 101/101 [==============================] - ETA: 0s - loss: 0.3681 - accuracy: 0.8702 Epoch 4: val_accuracy did not improve from 0.84570 101/101 [==============================] - 26s 260ms/step - loss: 0.3681 - accuracy: 0.8702 - val_loss: 0.4629 - val_accuracy: 0.8438 - lr: 0.0010 Epoch 5/10 101/101 [==============================] - ETA: 0s - loss: 0.3454 - accuracy: 0.8746 Epoch 5: val_accuracy improved from 0.84570 to 0.85156, saving model to flower_best_model.h5 101/101 [==============================] - 27s 264ms/step - loss: 0.3454 - accuracy: 0.8746 - val_loss: 0.4521 - val_accuracy: 0.8516 - lr: 0.0010 Epoch 6/10 101/101 [==============================] - ETA: 0s - loss: 0.3233 - accuracy: 0.8846 Epoch 6: val_accuracy improved from 0.85156 to 0.86523, saving model to flower_best_model.h5 101/101 [==============================] - 30s 293ms/step - loss: 0.3233 - accuracy: 0.8846 - val_loss: 0.4566 - val_accuracy: 0.8652 - lr: 0.0010 Epoch 7/10 101/101 [==============================] - ETA: 0s - loss: 0.3089 - accuracy: 0.8911 Epoch 7: val_accuracy did not improve from 0.86523 101/101 [==============================] - 32s 312ms/step - loss: 0.3089 - accuracy: 0.8911 - val_loss: 0.4549 - val_accuracy: 0.8652 - lr: 0.0010 Epoch 8/10 101/101 [==============================] - ETA: 0s - loss: 0.2912 - accuracy: 0.8945 Epoch 8: val_accuracy did not improve from 0.86523 101/101 [==============================] - 35s 347ms/step - loss: 0.2912 - accuracy: 0.8945 - val_loss: 0.4606 - val_accuracy: 0.8613 - lr: 0.0010 Epoch 9/10 101/101 [==============================] - ETA: 0s - loss: 0.2872 - accuracy: 0.9027 Epoch 9: val_accuracy did not improve from 0.86523 Epoch 9: ReduceLROnPlateau reducing learning rate to 0.0007000000332482159. 101/101 [==============================] - 34s 335ms/step - loss: 0.2872 - accuracy: 0.9027 - val_loss: 0.4731 - val_accuracy: 0.8613 - lr: 0.0010 Epoch 10/10 101/101 [==============================] - ETA: 0s - loss: 0.2641 - accuracy: 0.9114 Epoch 10: val_accuracy did not improve from 0.86523 101/101 [==============================] - 35s 342ms/step - loss: 0.2641 - accuracy: 0.9114 - val_loss: 0.4819 - val_accuracy: 0.8555 - lr: 7.0000e-04
def plot_history(history, loss_max=5):
"""
Check loss and accuracy evolution.
"""
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
fig, (ax1, ax2) = plt.subplots(1,2,figsize=(14, 4))
ax1.plot(acc, label='Training')
ax1.plot(val_acc, label='Validation')
ax1.legend(loc='lower right')
ax1.set(ylabel='Accuracy', title='Training - Validation Accuracy',
ylim=([min(plt.ylim()),1]))
ax2.plot(loss, label='Training')
ax2.plot(val_loss, label='Validation')
ax2.legend(loc='upper right')
ax2.set(ylabel='Loss (cross entropy)', xlabel='epochs',
title='Training - Validation Loss', ylim=([0, loss_max]))
plt.show()
plot_history(history, loss_max=1)
# Generator for test dataset
datagen = ImageDataGenerator(
rescale=1./255)
eval_datagen = datagen.flow(
Xtest, ytest,
batch_size=batch_size,
shuffle=False) # since shuffle was already during splitting into train, valid, test
# Evaluation on the test dataset
loss, acc = model.evaluate(eval_datagen, verbose=0)
print(f'Test loss: {loss:.2f}')
print(f'Test accuracy: {acc*100:.2f}%')
Test loss: 0.51 Test accuracy: 84.81%
It is now time for the fine tuning of our model: we're gonna unfreeze some of the top layers of the base model and train all those and the top layer classifier.
base_model.trainable = True
# Let's take a look to see how many layers are in the base model
print(f'Number of layers in the base model: {len(base_model.layers)}')
Number of layers in the base model: 154
# Fine-tune from this layer onwards
fine_tuning = 100
# Freeze all the layers before fine_tuned_ind
for layer in base_model.layers[:fine_tuning]:
layer.trainable = False
# Load best weights
# model.load_weights(weight_path)
# Finer learning rate now
opt = RMSprop(learning_rate=0.0001, rho=0.9, epsilon=1e-08, decay=0.0)
# Compilation
model.compile(optimizer=opt,
loss='categorical_crossentropy',
metrics=['accuracy'])
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
mobilenetv2_1.00_224 (Funct (None, 7, 7, 1280) 2257984
ional)
global_average_pooling2d (G (None, 1280) 0
lobalAveragePooling2D)
dense (Dense) (None, 5) 6405
=================================================================
Total params: 2,264,389
Trainable params: 1,867,845
Non-trainable params: 396,544
_________________________________________________________________
Continue training the model
fine_tuned_epochs = 10
total_epochs = epochs_0 + fine_tuned_epochs
history_fined = model.fit(
train_gen,
steps_per_epoch=len(Xtrain) // batch_size, # or batch_size=32
epochs=total_epochs,
initial_epoch=history.epoch[-1],
validation_data=valid_gen,
validation_steps=len(Xval) // batch_size,
callbacks=callbacks)
Epoch 10/20 101/101 [==============================] - ETA: 0s - loss: 0.3006 - accuracy: 0.8980 Epoch 10: val_accuracy did not improve from 0.86523 101/101 [==============================] - 39s 345ms/step - loss: 0.3006 - accuracy: 0.8980 - val_loss: 0.5133 - val_accuracy: 0.8652 - lr: 1.0000e-04 Epoch 11/20 101/101 [==============================] - ETA: 0s - loss: 0.1524 - accuracy: 0.9463 Epoch 11: val_accuracy improved from 0.86523 to 0.87109, saving model to flower_best_model.h5 101/101 [==============================] - 34s 335ms/step - loss: 0.1524 - accuracy: 0.9463 - val_loss: 0.5359 - val_accuracy: 0.8711 - lr: 1.0000e-04 Epoch 12/20 101/101 [==============================] - ETA: 0s - loss: 0.0878 - accuracy: 0.9704 Epoch 12: val_accuracy improved from 0.87109 to 0.88086, saving model to flower_best_model.h5 101/101 [==============================] - 34s 338ms/step - loss: 0.0878 - accuracy: 0.9704 - val_loss: 0.5252 - val_accuracy: 0.8809 - lr: 1.0000e-04 Epoch 13/20 101/101 [==============================] - ETA: 0s - loss: 0.0576 - accuracy: 0.9813 Epoch 13: val_accuracy did not improve from 0.88086 101/101 [==============================] - 34s 335ms/step - loss: 0.0576 - accuracy: 0.9813 - val_loss: 0.6044 - val_accuracy: 0.8750 - lr: 1.0000e-04 Epoch 14/20 101/101 [==============================] - ETA: 0s - loss: 0.0394 - accuracy: 0.9878 Epoch 14: val_accuracy improved from 0.88086 to 0.88672, saving model to flower_best_model.h5 101/101 [==============================] - 36s 353ms/step - loss: 0.0394 - accuracy: 0.9878 - val_loss: 0.5785 - val_accuracy: 0.8867 - lr: 1.0000e-04 Epoch 15/20 101/101 [==============================] - ETA: 0s - loss: 0.0305 - accuracy: 0.9885 Epoch 15: val_accuracy did not improve from 0.88672 101/101 [==============================] - 35s 344ms/step - loss: 0.0305 - accuracy: 0.9885 - val_loss: 0.7607 - val_accuracy: 0.8750 - lr: 1.0000e-04 Epoch 16/20 101/101 [==============================] - ETA: 0s - loss: 0.0260 - accuracy: 0.9916 Epoch 16: val_accuracy did not improve from 0.88672 101/101 [==============================] - 34s 333ms/step - loss: 0.0260 - accuracy: 0.9916 - val_loss: 0.7547 - val_accuracy: 0.8770 - lr: 1.0000e-04 Epoch 17/20 101/101 [==============================] - ETA: 0s - loss: 0.0216 - accuracy: 0.9931 Epoch 17: val_accuracy improved from 0.88672 to 0.89062, saving model to flower_best_model.h5 101/101 [==============================] - 34s 335ms/step - loss: 0.0216 - accuracy: 0.9931 - val_loss: 0.7266 - val_accuracy: 0.8906 - lr: 1.0000e-04 Epoch 18/20 101/101 [==============================] - ETA: 0s - loss: 0.0181 - accuracy: 0.9953 Epoch 18: val_accuracy did not improve from 0.89062 101/101 [==============================] - 33s 328ms/step - loss: 0.0181 - accuracy: 0.9953 - val_loss: 0.7616 - val_accuracy: 0.8848 - lr: 1.0000e-04 Epoch 19/20 101/101 [==============================] - ETA: 0s - loss: 0.0182 - accuracy: 0.9959 Epoch 19: val_accuracy improved from 0.89062 to 0.89453, saving model to flower_best_model.h5 101/101 [==============================] - 34s 332ms/step - loss: 0.0182 - accuracy: 0.9959 - val_loss: 0.6678 - val_accuracy: 0.8945 - lr: 1.0000e-04 Epoch 20/20 101/101 [==============================] - ETA: 0s - loss: 0.0150 - accuracy: 0.9953 Epoch 20: val_accuracy improved from 0.89453 to 0.90430, saving model to flower_best_model.h5 101/101 [==============================] - 35s 350ms/step - loss: 0.0150 - accuracy: 0.9953 - val_loss: 0.6614 - val_accuracy: 0.9043 - lr: 1.0000e-04
def plot_history_fined(history, history_fined, initial_epochs=epochs_0, loss_max=1):
"""
Check loss and accuracy evolution after fine tuning
"""
acc = history.history['accuracy'][:epochs_0]
acc += history_fined.history['accuracy']
val_acc = history.history['val_accuracy'][:epochs_0]
val_acc += history_fined.history['val_accuracy']
loss = history.history['loss'][:epochs_0]
loss += history_fined.history['loss']
val_loss = history.history['val_loss'][:epochs_0]
val_loss += history_fined.history['val_loss']
fig, (ax1, ax2) = plt.subplots(1,2,figsize=(14, 4))
ax1.plot(acc, label='Training')
ax1.plot(val_acc, label='Validation')
ax1.plot([initial_epochs-1,initial_epochs-1],
plt.ylim(), label='fine-tuning', ls='--')
ax1.legend(loc='lower right')
ax1.set(ylabel='Accuracy', title='Training - Validation Accuracy',
ylim=([0.4,1.005]))
ax2.plot(loss, label='Training')
ax2.plot(val_loss, label='Validation')
ax2.plot([initial_epochs-1,initial_epochs-1],
[0,1] , label='fine-tuning', ls='--')
ax2.legend(loc='upper right')
ax2.set(ylabel='Loss (cross entropy)', xlabel='epochs',
title='Training - Validation Loss', ylim=([0, loss_max]))
plt.show()
plot_history_fined(history, history_fined)
Great ! We can really see that fine-tuning is working and improve the accuracy of our model. We note also that the validation loss tends to increase a bit a the end: to prevent an eventual overfitting situation, we could add the EarlyStopping function in the callbacks during the training.
# Evaluation on the test dataset
loss, acc = model.evaluate(eval_datagen, verbose=0)
print(f'Test loss: {loss:.2f}')
print(f'Test accuracy: {acc*100:.2f}%')
Test loss: 0.52 Test accuracy: 90.74%
Indeed we have now an 90% accuracy on the test dataset (compared to 84% before fine tuning) !
import seaborn as sns
from sklearn import metrics
pred = model.predict(eval_datagen, verbose=1)
# get most likely class
y_pred = pred.argmax(axis=1)
y_true = ytest.argmax(axis=1)
print(metrics.classification_report(y_true, y_pred))
# confusion matrix
mat = metrics.confusion_matrix(y_true, y_pred)
df_mat = pd.DataFrame(mat, index=labels, columns=labels)
plt.figure(figsize=(8,6))
sns.heatmap(df_mat, annot=True, fmt='d', cmap=plt.cm.Reds)
#plt.tight_layout()
plt.ylabel('True label')
plt.xlabel('Predicted label')
17/17 [==============================] - 2s 71ms/step
precision recall f1-score support
0 0.96 0.86 0.90 99
1 0.90 0.95 0.92 128
2 0.89 0.87 0.88 99
3 0.93 0.93 0.93 88
4 0.88 0.92 0.90 126
accuracy 0.91 540
macro avg 0.91 0.91 0.91 540
weighted avg 0.91 0.91 0.91 540
Text(0.5, 36.72222222222221, 'Predicted label')
let's have a look at some predictions !
N = 20 # flowers to display
fig, axes = plt.subplots(4, 5, figsize=(20,12))
for i, ax in enumerate(axes.flat):
ax.imshow(Xtest[i].astype(np.uint8))
ax.set(xticks=[], yticks=[])
true = y_true[i]
prediction = y_pred[i]
ax.set_xlabel(f'Predict: {labels[prediction]}\n True: {labels[true]}',
color='black' if true == prediction else 'red')
#fig.tight_layout()
fig.suptitle('Predicted flowers; Incorrect Labels in Red', size=14)
Text(0.5, 0.98, 'Predicted flowers; Incorrect Labels in Red')
We can note the improvement of the model predictions by doing some fine-tuning. Of course, we can complexify the model by playing with the hyperparameters and/or adding other layers on top of the top-less MobileNetV2 base model such as several Dense layers with some Dropout or BatchNormalization ones between (to avoid overfitting). Feel free to test some different architectures to improve the accuracy of the predictions.
It will be interesting also to compare this fine-tuning method with others models such as VGG16, VGG19, ResNet50 etc.